Toward Distributed Simulation of Complex Discrete Event Systems Represented by Colored Petri Nets: A Review

determine statistics about the wait times in a queue. With standard PNs, it is di cult to calculate statistics based on individual ows, because tokens are all the same, and there is no order speci ed by which tokens are removed from a place. Since is it very useful to have this type of modeling power, a common HLPN extension is to allow the tokens in the system to be di erentiated by their type. This type of Petri net model is called a Colored Petri net (CPN) [19]. There have been a large number of variations and extensions of CPNs, many of which include ways to model time. Like PNs, CPNs were originally speci ed without time, and there are many di erent ways to integrate time into the models. Chiola [9] combined the CPN and GSPN models, whereas Jensen [20] adds time in a way that is compatible with Place-Transition (PrT) nets. Queuing Petri Nets More recently, Bause and Kemper [5, 7, 6] have combined the exibility of both queuing networks and PNs into a Queuing Petri net (QPN). The QPN model is an extension of CGSPNs which have the addition of timed queue places. Each queue place represents a single queue with an associated queue discipline, a set of servers, and a section for service completed. Tokens entering the queue place is not allowed to enable output transitions until it has been serviced by the servers in the place. In addition to the processing time speci ed by the places, QPNs also follow the notation of GSPNs by specifying delay times for transitions. Hence QPN models can be constructed to contain both pre-selection as well as race conditions. QPN models do o er much greater convenience, since the high level queuing places are used to represent a variety of complex queuing disciplines which may be di cult to create with a PN alone. It also makes it easier to change queuing disciplines within a model. 3.2 Petri Net Simulation Issues Although some forms of PNs are amenable to analytic solution, simulation is often the only way to achieve performance results of a system. One reason is that steady state solution of a Petri net involves the generation and solution of the entire state space. Models which exhibit the complexity of a a real system often have a state space which approaches or exceeds the limits of main memory in many of today's computers, making the calculation of this solution take much longer than reasonable. In a simulation, the reachability tree is not normally stored, and rarely solved, and instead statistics are gathered during the simulation runs. Since there are a limited amount of data to store, much larger systems can be analyzed with simulation than by other means. Another limitation of analytic solutions to PNs is that solution is limited to the class of PNs which are Markovian in nature. In many cases, this means that random events in the model must be approximated with exponential distributions. This approximation may introduce errors in the model which are intolerable. Simulation doesn't have such limitations. However, because of the diversity of problems to which PNs have been applied, a large number of di erent types of High-Level PNs have been developed. Many of these systems are incompatible with each other. The particular problem of compatibility is of concern when time is considered in the model. Since the original form of PNs had no notion of time, many di erent derivatives have been formed based on the original model. The most widely accepted type of PN is undoubtedly the Stochastic Petri net (SPN), which is the basis for much analysis. However, one of the most common forms of HLPN is the Colored Petri net (CPN), whose timed form is not directly compatible with the SPN, since it was derived more closely from Place Transition (PrT) nets. Although substantial work has been performed in analysis of CPNs, there has not been as much with timed CPNs. This con ict presents a di culty, since the modeler must choose between using low level modeling languages, such as GSPNs, which are more well understood, and the HLPNs, which allow more compact and manageable representation of their model at the sacri ce of modeling power. It is di cult to convert between the two because of the di erence in execution policies [1]. 3.2.1 Execution Policies When PNs were proposed by C. A. Petri [35], they strictly modeled the order of events, without modeling the time between events. Over the years, there have been several di erent methods for allowing time to be represented in a Petri net [36, 26, 28, 30]. Time has been added to both places and transitions, and these forms have been shown to be equivalent [36]. A more important issue with regard to time is the execution policy used in order to determine when and how transitions are red. Since a PN does not describe the order of ring of transitions which are enabled simultaneously, a transition must be chosen appropriately when time is involved. There are basically two schools of thought when it comes to choosing a transition among the set of enabled transitions: race and preselection. The choice is at least partially determined by the execution method of the PN. However, this discussion illustrates why the semantics of a PN may di er between several simulation or analysis tools. Race (or Firing Delay) One method of executing a timed PN is to use a method similar to untimed PNs in that each transition is red instantaneously. The token then remains in the input places of a transition until the time associated with this transition has expired. At this point in time, the transition is said to be \ready to re." Once it is chosen to re, tokens are removed from the input places, and tokens are immediately placed in the output places. This execution policy allows several transitions to be enabled by the same tokens, even though the ring of one transition will cause the other(s) to be disabled. In this case these transition are said to be competitive. This is the ring policy used for Generalized Stochastic Petri Nets (GSPNs) [4], where the transition to re is chosen probabilistically based on the ring rates of both transitions. This method of choice called a race or preemption policy, because it is equal to ring the transition whose time expires rst, preempting the ring of con icting transitions. Ciardo illustrated this procedure applied to SPNs with generally distributed ring delays in [11]. In addition to supporting immediate transitions, this procedure also supports the choice of resampling, enabling memory, or age memory [29] on a transition basis, allowing combinations to be used in a single model. The concept of a remaining ring time (RFT) is introduced, which indicates how long the transition must remain enabled before it is ready to re. After all of the RFTs are computed, the transition with the minimum RFT is chosen to re. It is red, and then the state is reevaluated, and transitions are resampled according to the speci ed policy. A timed Petri net where transitions are assigned exponentially distributed delays, is called a Stochastic Petri net (SPN). The memoryless property of the exponential distribution allows the entire state of the system to be speci ed by the marking alone, and so a CTMC can be built based on the transitions between states. When a timed Petri net contains transitions with distributions other than exponential, then the model cannot always be represented by a CTMC. This is because the state of the system includes not only the marking, but also the remaining ring time of the enabled transitions. However, there are speci c classes of SPNs with ring distributions other than exponential, which can still be analyzed [16, 11]. Pre-selection (or Firing Duration) Another method of executing time in a PN is called pre-selection. Under this policy, the tokens from the input places are removed when the transition is enabled and chosen to re. The tokens are then held in the transition until the time is expired, or placed in the output places with a time stamp equal to the ring time of the transition. In any case the tokens leaving the red transition are not allowed to enable any other transitions until the ring time has expired. In this form of execution, it is not obvious which transition should be chosen to re if they are con icting, since the times of the transition rings are not being considered. Even though Ajmone Marsan discussed pre-selection policies which make the net equivalent to a semi-Markov process [1], in practice this type of policy is very di cult to enforce with any reasonable size problem. For this reason, the authors observe that most tools which adhere to the pre-selection policy choose between competing transitions using some type of external policy. A popular choice is to give equal probability to all competing transitions. It is in the context of this fair policy that the term pre-selection will be used in the rest of this document. However, there are some simulators which have been observed to use unfair policies, essentially creating a priority of some transitions over others. Although this is a popular choice among simulation languages, it imposes external restrictions on the PN. Unfortunately, the choice between these two execution policies greatly a ects the semantics of the model. It is very di cult, if not impossible, to convert models between these two policies, and so it is necessary to know in advance what policy is being used when creating the model. Although it is accepted that this the policy is important to the semantics of the model [1], it is not generally stated which system is \better" from a modeling perspective. For this reason we have to determine a criteria for choosing one modeling system over another. The pre-selection policy (with fair selection) o ers simplicity, and therfore speed, to the creators of simulation tools. The race policy o ers simplicity to the modeler in that fewer sets of con icting transitions need to have explicitly stated probabilities, since the default is to allow con icts to race. Since many tools that use pre-selection don't allow reverse ring, the race policy gives the modeler the additional exibility of preempting a process after it starts and before it completes. To illustrate the modeling power of SPNs, Hass proved that any generalized semi-Markov Process has an equivalent SPN (with generally distributed ring times) [17]. 4 Distributed Simulation of Petri Nets Since PNs model asynchronous processes very well, it is reasonable to believe that distributed simulation of PN models would also perform admirably. The results that have been published in this area have been most encouraging, since they seem to indicate that PN simulations can indeed gain speed by distributing the problem. Both conservative and optimistic synchronization schemes have been applied to distributed PN simulation. Since PNs are often represented in their graph form, the most common type of partitioning is called Spatial Decomposition. This type of decomposition involves dividing the Petri net into a set of smaller isolated parts of the model, each of which is assigned to a processor for the duration of the simulation. Although this idea is simple because the partitioning only occurs once, it is often di cult to determine in advance which parts of the Petri net will be busy throughout the entire life of the simulation. This limitation causes an imbalance in the distribution of the simulation, and may drastically hurt performance. 4.1 Current Literature and Results There has been signi cant work in the area of distributed simulation of PNs in the past few years. Because PNs have unique properties in comparison to other types of simulation tools, there have been many attempts to take advantage of these properties. 4.1.1 Chiola and Ferscha Chiola and Ferscha [10] have applied their knowledge of SPNs and distributed simulation to develop a superior partitioning scheme. They furthered the idea of [33], who proposed that all output transitions of a decision place are contained in the same LP. This restriction was extended to include all the input places of transitions in structural con ict , in order to eliminate the need for interprocessor communication between a transition and input arcs in other LPs. They justi ably explain that this restriction is necessary to minimize communication load between processors. They have applied this partitioning scheme to both conservative and optimistic synchronization, and have shown that the balance of the partitioning is critical to the performance of the distributed simulation. They also concluded that using a very ne grained scheme like the minimum con ict region is not likely to give optimum performance because the processors are not kept busy. The propose that CPNs should provide enough complexity in the enabling combinations that distributed simulation should produce a better speedup over uniprocessor simulation. 4.1.2 Nicol et al. Nicol and Mao have produced the most complete published work on parallel simulation of PNs [31], building earlier work by Nicol and Roy [33]. Their work has achieved a speedup of about 10 on very large PNs (approximately 100,000 places and transitions), with a 16 processor parallel machine. These results are quite impressive, even though the models were of parallel machines, and therefore probably exhibited a natural good balance. The primary restrictions placed on partitioning were: a transition and its input places are always in the same LP, the tokens are committed at ring (pre-selection), and all output transitions have a non-zero ring time. The rst restriction guarantees that all members of a structural con ict remain in a single LP (as described in Section 4.1.1). The other restrictions of a non-zero ring time on output transitions of an LP was used along with pre-selection to produce the maximumlookahead for every message, since a conservative method was used. A detailed method is presented for relaxing the second restriction and allowing race conditions, but the authors anticipated that changing the model in this manner would cause a signi cant decrease in performance. Some ideas are presented to combating the problem, such as placing all output places of a transition into the same LP as the transition, or changing to an optimistic scheme to counteract the loss in anticipation. Most optimistic schemes used for distributed simulation of PNs with race wait until a transition is chosen to re before sending a message to output places which are contained in other LPs. In this paper, it is proposed that the opposite should occur, and that all competing transitions should be red tentatively, and cancel messages should be sent once a transition is chosen. Another important contribution is that the model has many LPs which are dynamically reassigned to other processors to accomplish load balancing during the simulation [32]. Every LP is simulated through a window of time, until they reach a global synchronization point. At that time, the decision is made on whether or not to remap LPs between processors. When it is determined that there are processors which have too heavy of a load, some LPs are moved to processors which are under less load. 4.1.3 Nketsa Nketsa developed methods for applying conservative simulation techniques to PNs [34]. Communication among partitions is handled by asynchronous communication, or by having common places shared between two subnets. A simulation algorithm is presented which reduces the amount of null messages as well as detecting deadlocks which result from the use of the conservative simulation algorithm. They propose to reduce the null messages by using reduction rules to determine the shortest paths through an LP, and therefore be able to quickly evaluate the earliest possible output messages for the LP. The author claims that this prediction would allow processes to proceed further than the standard conservative method. Results are not yet presented. 4.1.4 Kumar and Harous Another way to apply conservative simulation to PNs which has been discussed is to use a modi ed form of Timed Petri Nets (TPN) [24]. In this model, each place and transition is composed of an instantaneous input process, a queue, and an instantaneous output process. An approach is presented to synchronize transitions in structural con ict if they are placed in di erent LPs, allowing any partitioning. However, in the paper, they restrict the discussion to the case were each sub process of a place or transition is an LP. Ideas are also presented whichwould reduce null messages and avoid deadlock based on the properties of PNs,by modifying and extending [8]. NULL messages are reduced by consideringnot the minimum messages, but the minimum time that a transition can re.In the case of a transition with multiple input places, the NULL message can beincreased to the maximum time on the input places, since the transition cannotbe red until all input is available.4.1.5 Ammar and DengTime Warp was successfully applied to stochastic Petri nets by Ammar andDeng [2]. In their implementation, partitioning is done by cutting arcs, andcreating duplicate places for those arcs in each of the corresponding processes.There are no restrictions on which arcs may be cut, nor which places and tran-sitions reside together in the same subnet. The optimistic nature of Time Warpis used to allow the ring of a transition even if the state of the input transitionsis unknown [3]. Although it was demonstrated that the Time Warp simulationproduced results that agreed with GreatSPN, it is not apparent what level ofspeedup was achieved by distributing the simulation. However, it is probablethat the exible nature of the partitioning would make it very di cult to balancethe system.In addition to using Time Warp and arbitrary arc cutting, Ammar [2] hasproposed what is called Time Scale Decomposition (TSD) to aid in partitioninga PN being simulated with the Time Warp algorithm. TSD is performed byremoving slow subnets (rare events) from the simulation in order to producea more balanced model. The results from the simulation are then placed backinto a model which contains the rare events, and the steady state solution iscalculated. Although this method may prove di cult to automate, it is anothertool that can be used to balance a distributed simulation.4.1.6 SchofThere is one known distributed simulator which can handle a form of high-levelnets called Timed Hierarchical Object-Related Nets (THORNs) [37]. Transi-tions in the THORN models have enabling, action, delay time and ring timefunctions which are written in C++. Due to the amount of exibility in aTHORN model, and the resulting di culty of predicting future events, the dis-tributed simulator uses Time Warp. Speedup was moderate, with a speedup ofabout 5.7 on 6 processors with a hand tuned model, and about 3.5 on 6 proces-sors with a heuristic based partitioning. However, this was the only distributedsystem reviewed with very large processor/communication speed ratio.4.1.7 Lin and TuThere has also been at least some work in re-de ning timed Petri nets in order toavoid the use of NULL messages or of rollback altogether. In [25], a Timed PetriNet with Firing Durations (TPNFD) is de ned, in which a transition may notre even though it is enabled for its speci ed delay. Although this does providea way to increase the speed of simulation, it also presents other problems. Sincea transition is not required to re if it is enabled before a competing transition, the choice of which transition to re is a ected by the simulation process itself.This means that simulating the net on one distributed system may yield di erentresults than another, which is usually an undesirable a ect.5 Future DirectionsIt can be inferred from the volume of research in the area of Petri net simulationthat there are good reasons for using a robust system like PNs in order to createsimulationmodels. It has been argued that HLPNs (e.g. Colored Petri nets) arean e ective way to utilize the power of PN models for large systems, by usinghigher level constructs, at the price of a decrease in e ciency. This decrease ine ciency motivates us to believe that the power of distributed simulation is ab-solutely critical if high level models are to be simulated. The research reviewedhas made it clear that in order to e ectively apply distributed simulation toPNs the LPs should be composed of relatively complex subnets, with hundredsof places and transitions. Since components of a CPN are much more complex,it is expected that a smaller number of nodes per LP will be necessary. How-ever, solutions to the partitioning problem for PNs will have to be modi ed tosupport CPNs e ectively, especially if the CPN model supports the concept ofpreemption.Preemption is a powerful feature in many PN models, including manufactur-ing and tra c control. There are many advantages of these types of constructs insimulation. However, there appears to be little research to produce distributedsimulator which could both be used to model preemption while producing rea-sonable increase in speed over a sequential simulation. The most likely reason forthis is that the race condition makes prediction of the future di cult, a ectingthe amount of look-ahead possible for conservative simulations.Both conservative and optimistic simulation depend on the ability of the LPto e ectively predict the result of the simulation in the future. Most optimisticsimulation schemes allow the LP to continue on the basis that it will receiveno messages with an earlier time stamp, because if this information had beenavailable, then the simulation may have proceeded di erently. If a message isreceived that meets this criteria, then the LP is rolled back to the time stampon the newly arrived message. Since too much roll back is undesireable, itis necessary to develop ways to more e ectively predict future events. Thisproblem is analogous to that of branch prediction in microprocessors. Ratherthan always allowing LPs to simulate based on the same assumptions (that nomessages will be received), the state and structure of the PN could be utilizedto predict the most likely events. It is believed that this would produce a moree ective framework for distributed simulation of CPNs.References[1] M. Ajmone Marsan, G. Balbo, A. Bobbio, G. Chiola, G. Conte, andA. Culmani. The e ect of execution policies on the semantics and analy-sis of stochastic petri nets. IEEE Transactions on Software Engineering,15(7):832{845, Jul 1989. [2] H. H. Ammar and S. Deng. Time warp simulation of stochastic petri nets. InPetri Nets and Performance Models, pages 186{195,Melbourne, Autstralia,Dec 1991. IEEE.[3] H. H. Ammar and S. Deng. Time warp simulation using time scale de-composition. ACM Transactions on Modeling and Computer Simulation(TOMACS), 2(2):158{177, Apr 1992.[4] G. Balbo, M. AjmoneMarsan, and G. Conte. A class of generalized stochas-tic petri nets for the performance evaluation of multiprocessor systems.ACM, 2(2):93{122, May 1984.[5] F. Bause. Queueing Petri Nets: A Formalism for the Combined Qualitativeand Quantitative Analysis of Systems. In 5th International Workshop onPetri Nets and Performance Models, Toulouse (France), pages 14{23, Oct1993.[6] F. Bause, P. Buchholz, and P. Kemper. Hierarchically combined QueueingPetri Nets. In 11th International Conference on Analysis and Optimizationsof Systems, Discrete Event Systems, Sophia-Antipolis (France), June 1994.[7] F. Bause and P. Kemper. QPN-Tool for the qualitative and quantitativeanalysis of Queueing Petri Nets. In G. Haring and G. Kotsis, editors, Proc.of the 7th International Conference on Computer Performance Evaluation,Modelling Techniques and Tools, Vienna (Austria), LNCS 794, pages 321{334. Springer-Verlag, Berlin, 1994.[8] K. M. Chandy and J. Mirsa. Distributed simulation: A case study in designand veri cation of distributed programs. IEEE Transactions on SoftwareEngineering, SE-5(5):440{452, Sep 1979.[9] G. Chiola, G. Bruno, and T. Demaria. Introducing a color formalism intogeneralized stochastic petri nets. In Proceedings of the 9th European Work-shop on Application and Theory of Petri Nets, Venezia, Italy, 1988.[10] G. Chiola and A. Ferscha. Distributed simulation of petri nets. IEEEParallel & Distributed Technology, 1(3):33{49, Aug 1993.[11] G. Ciardo, R. German, and C. Lindemann. A characterization of thestochastic process underlying a stochastic petri net. IEEE Transactionson Software Engineering, 20(7):506{515, July 1994.[12] F. DiCesare, M. R. Gile, and P. T. Kulp. A code generation tool for sim-ulation and control, based on colored petri nets. In IEEE InternationalConference on Systems, Man and Cybernetics, pages 3063{3068, Vancou-ver, BC, Can, oct 1995.[13] F. DiCesare, R. Graves, and M. R. Gile. Petri nets and interactive multi-media in manufacturing systems education. In Computational Engineeringin Systems Applications, pages 13{18, Lille; France, July 1996.[14] F. DiCesare, P. T. Kulp, M. R. Gile, and G. List. The application ofpetri nets to the modeling, analysis and control of intelligent urban tra cnetworks. In R. Valette, editor, Application and theory of Petri nets, volume 815 of Lecture Notes in Computer Science, pages 2{15. Springer-Verlag,Zaragoza; Spain, Jun 1994.[15] F. DiCesare, G. List, P. Kulp, T. Myers, and M. R. Gile. Petri Net Model-ing and Simulation for Intelligent Control of Tra c Networks. In Compu-tational Engineering in Systems Applications, pages 13{18, Lille; France,July 1996.[16] J. B. Dugan, K. S. Trivedi, R. M. Geist, and V. F. Nicola. Extendedstochastic petri nets: Applications and analysis. Performance '84: Modelsof Computer System Performance, Proceedings of the Tenth InternationalSymposium, 1985.[17] P. J. Haas and G. S. Shedler. Stochastic petri net representation of discreteevent simulations. IEEE Transactions on Software Engineering, 15(4):381{393, Apr 1989.[18] D. R. Je erson and H. A. Sowizral. Fast Concurrent Simulation using theTime Warp Mechanism, Part I: Local Control. Technical report, The RandCorp., Santa Monica, CA, Dec 1982.[19] K. Jensen. Coloured petri nets and the invariant method. TheoreticalComputer Science, 15:317{336, 1981.[20] K. Jensen. Coloured Petri nets : basic concepts, analysis methods, andpractical use. Springer-Verlag, Berlin ; New York, 1992.[21] K. Jensen and G. Rozenberg, editors. High-level Petri nets: theory andapplication. Springer-Verlag, Berlin ; New York, 1991.[22] D. Kumar. Systems with low distributed overhead. IEEE Trans. on Parallel& Distributed Systems, 3(2), 1992.[23] D. Kumar and S. Harous. A Study of Achievable Speedup in DistributedSimulation via NULL Messages. IEEE Transactions on Parallel and Dis-tributed Systems, 4(3):347{354, 1993.[24] D. Kumar and S. Harous. Distributed simulation of timed petri nets: Basicproblems and their resolution. IEEE Conference on Systems Man andCybernetics, 24(10), 1994.[25] Y. Lin and S. Tu. Distributed simulation of timed petri nets modelingdistributed systems. In 37th Midwest symposium Circuits and systems,Aug 1994.[26] P. M. Merlin and D. J. Farber. Recoverability of communication protocols {implication of a theoretical study. IEEE Transactions on Communications,25(9):1036{1043, 1976.[27] J. Mirsa. Distributed discrete-event simulation. Computing Surveys, 18(1),March 1986.[28] M. K. Molloy. On the Integration of Delay and Throughput Measures inDistributed Processing Systems. PhD thesis, University of California, LosAngeles, CA; USA, 1981. [29] T. Murata. Petri nets: Properties, analysis and applications. Proceedingsof the IEEE, 77(4):541{580, 1989.[30] S. Natkin. Les r eseaux de Petri Stochastiques. PhD thesis, CNAM, Paris,June 1980. th ese de docteur ing enieur.[31] D. M. Nicol and W. Mao. Automated parallelization of timed petri-netsimulations. Journal of Parallel and Distributed Computing, 29(1), Aug1995.[32] D. M. Nicol and J. P. F. Reynolds. Optimal dynamic remapping of dataparallel computations. IEEE Transactions on Computers, 40(3):295{306,1991.[33] D. M. Nicol and S. Roy. Parallel simulation of timed petri-nets. In Pro-ceedings of the 1991 Winter Simulation Conference, pages 574{583, 1991.[34] A. Nketsa. Conervative distributed simulation of petri nets. Computers &Mathematics with Applications, May 1994.[35] C. A. Petri. Kommunikation mit Automaten. PhD thesis, Bonn: Institutfur Instrumentelle Mathematik, Schriften des IIM Nr. 3, 1962.[36] C. Ramchandani. Analysis of Asynchronous Concurrent Systems by TimedPetri nets. PhD thesis, MIT, 1973.[37] S. Schof. A distributed simulation engine for hierarchical petri nets. In Sim-ulation verteilter Systeme und paralleler Prozesse, pages 153{159, Dresden,Germany, October 1995.